Tell us about your PDF experience.

Azure OpenAI Service Documentation

Learn how to use Azure OpenAI's powerful language models including the GPT-3, Codex and Embeddings model series for content generation, summarization, semantic search, and natural language to code translation.

O V E R V I E W

What is Azure OpenAI Service?

Q U I C K S T A R T

Quickstarts

H O W - T O G U I D E

Create a resource

T U T O R I A L

Embeddings

H O W - T O G U I D E

Completions

T R A I N I N G

Intro to Azure OpenAI training

C O N C E P T

Azure OpenAI Models

R E F E R E N C E

Support and help options

Additional resources

Azure OpenAI

Azure OpenAI Studio Region support Quotas and limits

Apply for access to Azure OpenAI

Video

Combining OpenAI models with the power of Azure

Reference

REST API

Terms of use

Tools

Azure CLI PowerShell

What is Azure OpenAI Service?

Article • 05/01/2023

Azure OpenAI Service provides REST API access to OpenAI's powerful language models including the GPT-3, Codex and Embeddings model series. In addition, the new GPT-4 and ChatGPT (gpt-35-turbo) model series are now available in preview. These models can be easily adapted to your specific task including but not limited to content generation, summarization, semantic search, and natural language to code translation. Users can access the service through REST APIs, Python SDK, or our web-based interface in the Azure OpenAI Studio.

Features overview

Responsible AI

At Microsoft, we're committed to the advancement of AI driven by principles that put people first. Generative models such as the ones available in Azure OpenAI have significant potential benefits, but without careful design and thoughtful mitigations, such models have the potential to generate incorrect or even harmful content. Microsoft has made significant investments to help guard against abuse and unintended harm, which includes requiring applicants to show well-defined use cases, incorporating Microsoft’s principles for responsible AI use , building content filters to support customers, and providing responsible AI implementation guidance to onboarded customers.

How do I get access to Azure OpenAI?

How do I get access to Azure OpenAI?

Access is currently limited as we navigate high demand, upcoming product improvements, and Microsoft’s commitment to responsible AI . For now, we're working with customers with an existing partnership with Microsoft, lower risk use cases, and those committed to incorporating mitigations.

More specific information is included in the application form. We appreciate your patience as we work to responsibly enable broader access to Azure OpenAI.

Apply here for access:

Apply now

Comparing Azure OpenAI and OpenAI

Azure OpenAI Service gives customers advanced language AI with OpenAI GPT-4, GPT-3, Codex, and DALL-E models with the security and enterprise promise of Azure. Azure OpenAI co-develops the APIs with OpenAI, ensuring compatibility and a smooth transition from one to the other.

With Azure OpenAI, customers get the security capabilities of Microsoft Azure while running the same models as OpenAI. Azure OpenAI offers private networking, regional availability, and responsible AI content filtering.

Key concepts

Prompts & Completions

The completions endpoint is the core component of the API service. This API provides access to the model's text-in, text-out interface. Users simply need to provide an input prompt containing the English text command, and the model will generate a text completion.

Here's an example of a simple prompt and completion:

""" count to 5 in a for loop """

Prompt:

for i in range(1, 6): print(i)

Completion:

Tokens

Azure OpenAI processes text by breaking it down into tokens. Tokens can be words or just chunks of characters. For example, the word “hamburger” gets broken up into the tokens “ham”, “bur” and “ger”, while a short and common word like “pear” is a single token. Many tokens start with a whitespace, for example “ hello” and “ bye”.

The total number of tokens processed in a given request depends on the length of your input, output and request parameters. The quantity of tokens being processed will also affect your response latency and throughput for the models.

Resources

Azure OpenAI is a new product offering on Azure. You can get started with Azure OpenAI the same way as any other Azure product where you create a resource, or instance of the service, in your Azure Subscription. You can read more about Azure's resource management design.

Deployments

Once you create an Azure OpenAI Resource, you must deploy a model before you can start making API calls and generating text. This action can be done using the

Deployment APIs. These APIs allow you to specify the model you wish to use.

In-context learning

The models used by Azure OpenAI use natural language instructions and examples provided during the generation call to identify the task being asked and skill required. When you use this approach, the first part of the prompt includes natural language instructions and/or examples of the specific task desired. The model then completes the task by predicting the most probable next piece of text. This technique is known as "in-context" learning. These models aren't retrained during this step but instead give predictions based on the context you include in the prompt.

There are three main approaches for in-context learning: Few-shot, one-shot and zero-shot. These approaches vary based on the amount of task-specific data that is given to the model:

Few-shot: In this case, a user includes several examples in the call prompt that demonstrate the expected answer format and content. The following example shows a few-shot prompt where we provide multiple examples (the model will generate the last answer):

The number of examples typically range from 0 to 100 depending on how many can fit in the maximum input length for a single prompt. Maximum input length can vary

depending on the specific models you use. Few-shot learning enables a major reduction in the amount of task-specific data required for accurate predictions. This approach will typically perform less accurately than a fine-tuned model.

One-shot: This case is the same as the few-shot approach except only one example is provided.

Zero-shot: In this case, no examples are provided to the model and only the task request is provided.

Models

The service provides users access to several different models. Each model provides a different capability and price point.

GPT-4 models are the latest available models. These models are currently in preview. For access, existing Azure OpenAI customers can apply by filling out this form .

The GPT-3 base models are known as Davinci, Curie, Babbage, and Ada in decreasing order of capability and increasing order of speed.

The Codex series of models is a descendant of GPT-3 and has been trained on both natural language and code to power natural language to code use cases. Learn more about each model on our models concept page.

Next steps

Learn more about the underlying models that power Azure OpenAI.

Azure OpenAI Service quotas and limits

Article • 04/27/2023

This article contains a quick reference and a detailed description of the quotas and limits for Azure OpenAI in Azure Cognitive Services.

Quotas and limits reference

The following sections provide you with a quick guide to the quotas and limits that apply to the Azure OpenAI:

*The limits are subject to change. We anticipate that you will need higher limits as you move toward production and your solution scales. When you know your solution requirements, please reach out to us by applying for a quota increase here: https://aka.ms/oai/quotaincrease

For information on max tokens for different models, consult the models article

General best practices to mitigate throttling during autoscaling

To minimize issues related to throttling, it's a good idea to use the following techniques: Implement retry logic in your application.

Avoid sharp changes in the workload. Increase the workload gradually. Test different load increase patterns.

Create another OpenAI service resource in the same or different regions, and distribute the workload among them.

The next sections describe specific cases of adjusting quotas.

How to request increases to the default quotas and limits

At this time, due to overwhelming demand we cannot accept any new resource or quota increase requests.

７ Note

Ensure that you thoroughly assess your current resource utilization, approaching its full capacity. Be aware that we will not grant additional resources if efficient usage of existing resources is not observed.

Next steps

Learn more about the underlying models that power Azure OpenAI.

Azure OpenAI Service models

Article • 05/11/2023

Azure OpenAI provides access to many different models, grouped by family and capability. A model family typically associates models by their intended task. The following table describes model families currently available in Azure OpenAI. Not all models are available in all regions currently. Refer to the model capability table in this article for a full breakdown.

Model capabilities

Each model family has a series of models that are further distinguished by capability. These capabilities are typically identified by names, and the alphabetical order of these names generally signifies the relative capability and cost of that model within a given model family. For example, GPT-3 models use names such as Ada, Babbage, Curie, and Davinci to indicate relative capability and cost. Davinci is more capable and more expensive than Curie, which in turn is more capable and more expensive than Babbage, and so on.

７ Note

Any task that can be performed by a less capable model like Ada can be performed by a more capable model like Curie or Davinci.

Naming convention

Azure OpenAI model names typically correspond to the following standard naming convention:

{capability}-{family}[-{input-type}]-{identifier}

For example, our most powerful GPT-3 model is called text-davinci-003 , while our most powerful Codex model is called code-davinci-002 .

davinci

The older versions of GPT-3 models named ada , babbage , curie , and that

don't follow the standard naming convention are primarily intended for fine tuning. For more information, see Learn how to customize a model for your application.

Finding what models are available

You can get a list of models that are available for both inference and fine-tuning by your Azure OpenAI resource by using the Models List API.

Finding the right model

We recommend starting with the most capable model in a model family to confirm whether the model capabilities meet your requirements. Then you can stay with that model or move to a model with lower capability and cost, optimizing around that model's capabilities.

GPT-4 models (preview)

GPT-4 can solve difficult problems with greater accuracy than any of OpenAI's previous models. Like gpt-35-turbo, GPT-4 is optimized for chat but works well for traditional completions tasks.

These models are currently in preview. For access, existing Azure OpenAI customers can apply by filling out this form .

gpt-4

gpt-4-32k

gpt-4

The tokens.

supports 8192 max input tokens and the

supports up to 32,768

gpt-4-32k

GPT-3 models

The GPT-3 models can understand and generate natural language. The service offers four model capabilities, each with different levels of power and speed suitable for different tasks. Davinci is the most capable model, while Ada is the fastest. In the order of greater to lesser capability, the models are:

text-davinci-003

text-curie-001

text-babbage-001

text-ada-001

While Davinci is the most capable, the other models provide significant speed advantages. Our recommendation is for users to start with Davinci while experimenting, because it produces the best results and validate the value that Azure OpenAI can provide. Once you have a prototype working, you can then optimize your model choice with the best latency/performance balance for your application.

Davinci

Davinci is the most capable model and can perform any task the other models can perform, often with less instruction. For applications requiring deep understanding of the content, like summarization for a specific audience and creative content generation, Davinci produces the best results. The increased capabilities provided by Davinci require more compute resources, so Davinci costs more and isn't as fast as other models.

Another area where Davinci excels is in understanding the intent of text. Davinci is excellent at solving many kinds of logic problems and explaining the motives of

characters. Davinci has been able to solve some of the most challenging AI problems involving cause and effect.

Use for: Complex intent, cause and effect, summarization for audience

Curie

Curie is powerful, yet fast. While Davinci is stronger when it comes to analyzing complicated text, Curie is capable for many nuanced tasks like sentiment classification and summarization. Curie is also good at answering questions and performing Q&A and as a general service chatbot.

Use for: Language translation, complex classification, text sentiment, summarization

Babbage

Babbage can perform straightforward tasks like simple classification. It’s also capable when it comes to semantic search, ranking how well documents match up with search queries.

Use for: Moderate classification, semantic search classification

Ada

Ada is usually the fastest model and can perform tasks like parsing text, address correction and certain kinds of classification tasks that don’t require too much nuance. Ada’s performance can often be improved by providing more context.

Use for: Parsing text, simple classification, address correction, keywords

ChatGPT (gpt-35-turbo) (preview)

The ChatGPT model (gpt-35-turbo) is a language model designed for conversational interfaces and the model behaves differently than previous GPT-3 models. Previous models were text-in and text-out, meaning they accepted a prompt string and returned a completion to append to the prompt. However, the ChatGPT model is conversation-in and message-out. The model expects a prompt string formatted in a specific chat-like transcript format, and returns a completion that represents a model-written message in the chat.

To learn more about the ChatGPT model and how to interact with the Chat API check out our in-depth how-to.

Codex models

The Codex models are descendants of our base GPT-3 models that can understand and generate code. Their training data contains both natural language and billions of lines of public code from GitHub.

They’re most capable in Python and proficient in over a dozen languages, including C#, JavaScript, Go, Perl, PHP, Ruby, Swift, TypeScript, SQL, and Shell. In the order of greater to lesser capability, the Codex models are:

code-davinci-002

code-cushman-001

Davinci

Similar to GPT-3, Davinci is the most capable Codex model and can perform any task the other models can perform, often with less instruction. For applications requiring deep understanding of the content, Davinci produces the best results. Greater capabilities require more compute resources, so Davinci costs more and isn't as fast as other models.

Cushman

Cushman is powerful, yet fast. While Davinci is stronger when it comes to analyzing complicated tasks, Cushman is a capable model for many code generation tasks.

Cushman typically runs faster and cheaper than Davinci, as well.

Embeddings models

） Important

We strongly recommend using text-embedding-ada-002 (Version 2) . This model/version provides parity with OpenAI's text-embedding-ada-002 . To learn more about the improvements offered by this model, please refer to OpenAI's blog post . Even if you are currently using Version 1 you should migrate to Version 2 to take advantage of the latest weights/updated token limit. Version 1 and Version 2 are not interchangeable, so document embedding and document search must be done using the same version of the model.

Currently, we offer three families of Embeddings models for different functionalities:

Similarity Text search Code search

Each family includes models across a range of capability. The following list indicates the length of the numerical vector returned by the service, based on model capability:

Ada: 1024 dimensions

Babbage: 2048 dimensions

Curie: 4096 dimensions

Davinci: 12288 dimensions

Davinci is the most capable, but is slower and more expensive than the other models. Ada is the least capable, but is both faster and cheaper.

Similarity embedding

These models are good at capturing semantic similarity between two or more pieces of text.

Text search embedding

These models help measure whether long documents are relevant to a short search query. There are two input types supported by this family: doc , for embedding the documents to be retrieved, and query , for embedding the search query.

Code search embedding

Similar to text search embedding models, there are two input types supported by this family: code , for embedding code snippets to be retrieved, and text , for embedding natural language search queries.

When using our embeddings models, keep in mind their limitations and risks.

Model Summary table and region availability

） Important

South Central US is temporarily unavailable for creating new resources due to high demand.

GPT-3 Models

These models can be used with Completion API requests.

gpt-35-turbo

is the only

model that can be used with both Completion API requests and the Chat Completion API.

0301

1 Currently, only version of this model is available. This version of the model will be

deprecated on 8/1/2023 in favor of newer version of the gpt-35-model. See ChatGPT model versioning for more details.

GPT-4 Models

These models can only be used with the Chat Completion API.

1 The model is in preview and only available by request .

0314

2 Currently, only version of this model is available.

Codex Models

These models can only be used with Completions API requests.

1 The model is available for fine-tuning by request only. Currently we aren't accepting new requests to fine-tune the model.

Embeddings Models

These models can only be used with Embedding API requests.

７ Note

We strongly recommend using text-embedding-ada-002 (Version 2) . This model/version provides parity with OpenAI's text-embedding-ada-002 . To learn more about the improvements offered by this model, please refer to OpenAI's blog post . Even if you are currently using Version 1 you should migrate to Version 2 to take advantage of the latest weights/updated token limit. Version 1 and Version 2 are not interchangeable, so document embedding and document search must be done using the same version of the model.

Next steps

Learn more about Azure OpenAI

Learn more about fine-tuning Azure OpenAI models

What's new in Azure OpenAI Service

Article • 05/11/2023

May 2023

Azure OpenAI is now available in the France Central region. Check the models page, for the latest information on model availability in each region.

April 2023

Inactive deployments of customized models will now be deleted after 15 days; models will remain available for redeployment. If a customized (fine-tuned) model is deployed for more than fifteen (15) days during which no completions or

chat completions calls are made to it, the deployment will automatically be deleted (and no further hosting charges will be incurred for that deployment). The underlying customized model will remain available and can be redeployed at any time. To learn more check out the how-to-article.

March 2023

GPT-4 series models are now available in preview on Azure OpenAI. To request access, existing Azure OpenAI customers can apply by filling out this form . These models are currently available in the East US and South Central US regions.

New Chat Completion API for ChatGPT and GPT-4 models released in preview on 3/21. To learn more checkout the updated quickstarts and how-to article.

ChatGPT (gpt-35-turbo) preview. To learn more checkout the how-to article.

Increased training limits for fine-tuning: The max training job size (tokens in training file) x (# of epochs) is 2 Billion tokens for all models. We have also increased the max training job from 120 to 720 hours.

Adding additional use cases to your existing access. Previously, the process for adding new use cases required customers to reapply to the service. Now, we're releasing a new process that allows you to quickly add new use cases to your use of the service. This process follows the established Limited Access process within Azure Cognitive Services. Existing customers can attest to any and all new use cases here . Please note that this is required anytime you would like to use the service for a new use case you did not originally apply for.

February 2023

New Features

.NET SDK(inference) preview release | Samples

suffix

Terraform SDK update to support Azure OpenAI management operations. Inserting text at the end of a completion is now supported with the parameter.

Updates

Content filtering is on by default.

New articles on:

Monitoring an Azure OpenAI Service Plan and manage costs for Azure OpenAI

New training course:

Intro to Azure OpenAI

January 2023

New Features

Service GA. Azure OpenAI Service is now generally available.

New models: Addition of the latest text model, text-davinci-003 (East US, West Europe), text-ada-embeddings-002 (East US, South Central US, West Europe)

December 2022

New features

The latest models from OpenAI. Azure OpenAI provides access to all the latest models including the GPT-3.5 series.

New API version (2022-12-01). This update includes several requested enhancements including token usage information in the API response, improved

error messages for files, alignment with OpenAI on fine-tuning creation data structure, and support for the suffix parameter to allow custom naming of fine-tuned jobs.

Higher request per second limits. 50 for non-Davinci models. 20 for Davinci models.

Faster fine-tune deployments. Deploy an Ada and Curie fine-tuned models in under 10 minutes.

Higher training limits: 40M training tokens for Ada, Babbage, and Curie. 10M for Davinci.

Process for requesting modifications to the abuse & miss-use data logging & human review. Today, the service logs request/response data for the purposes of abuse and misuse detection to ensure that these powerful models aren't abused. However, many customers have strict data privacy and security requirements that require greater control over their data. To support these use cases, we're releasing a new process for customers to modify the content filtering policies or turn off the abuse logging for low-risk use cases. This process follows the established Limited Access process within Azure Cognitive Services and existing OpenAI customers can apply here .

Customer managed key (CMK) encryption. CMK provides customers greater control over managing their data in Azure OpenAI by providing their own encryption keys used for storing training data and customized models. Customer-managed keys (CMK), also known as bring your own key (BYOK), offer greater flexibility to create, rotate, disable, and revoke access controls. You can also audit the encryption keys used to protect your data. Learn more from our encryption at rest documentation.

Lockbox support SOC-2 compliance

Logging and diagnostics through Azure Resource Health, Cost Analysis, and Metrics & Diagnostic settings.

Studio improvements. Numerous usability improvements to the Studio workflow including Azure AD role support to control who in the team has access to create fine-tuned models and deploy.

Changes (breaking)

Fine-tuning create API request has been updated to match OpenAI’s schema.

Preview API versions:

API version 2022-12-01:

Content filtering is temporarily off by default. Azure content moderation works differently than OpenAI. Azure OpenAI runs content filters during the generation call to detect harmful or abusive content and filters them from the response. Learn More

These models will be re-enabled in Q1 2023 and be on by default.

Customer actions

Contact Azure Support if you would like these turned on for your subscription. Apply for filtering modifications , if you would like to have them remain off. (This option will be for low-risk use cases only.)

Next steps

Learn more about the underlying models that power Azure OpenAI.

Azure OpenAI Service frequently asked questions

FAQ

If you can't find answers to your questions in this document, and still need help check the Cognitive Services support options guide. Azure OpenAI is part of Azure Cognitive Services.

Data and Privacy

Do you use my company data to train any of the models?

Azure OpenAI doesn't use customer data to retrain models. For more information, see the Azure OpenAI data, privacy, and security guide.

General

Does Azure OpenAI support GPT-4?

Azure OpenAI supports the latest GPT-4 models. These models are currently only available by request. For access, existing Azure OpenAI customers can apply by filling out this form .

How do the capabilities of Azure OpenAI compare to OpenAI?

Azure OpenAI Service gives customers advanced language AI with OpenAI GPT-3, Codex, and DALL-E models with the security and enterprise promise of Azure. Azure OpenAI co-develops the APIs with OpenAI, ensuring compatibility and a smooth transition from one to the other.

With Azure OpenAI, customers get the security capabilities of Microsoft Azure while running the same models as OpenAI.

Does Azure OpenAI support VNETs and Private Endpoints?

Yes, as part of Azure Cognitive Services, Azure OpenAI supports VNETs and Private Endpoints. To learn more consult the Cognitive Services virtual networking guidance

Do the GPT-4 models currently support image input?

No, GPT-4 is designed by OpenAI to be multimodal, but currently only text input and output are supported.

How do I apply for new use cases?

Previously, the process for adding new use cases required customers to reapply to the service. Now, we're releasing a new process that allows you to quickly add new use cases to your use of the service. This process follows the established Limited Access process within Azure Cognitive Services. Existing customers can attest to any and all new use cases here . Please note that this is required anytime you would like to use the service for a new use case you did not originally apply for.

I am trying to use embeddings and received the error "InvalidRequestError: Too many inputs. The max number of inputs is 1." How do I fix this?

This error typically occurs when you try to send a batch of text to embed in a single API request as an array. Currently Azure OpenAI does not support batching with embedding requests. Embeddings API calls should consist of a single string input per request. The string can be up to 8191 tokens in length when using the text-embedding-ada-002 (Version 2) model.

Where can I read about better ways to use Azure OpenAI to get the responses I want from the service?

Check out our introduction to prompt engineering While these models are extremely powerful, their behavior is also very sensitive to the prompts they receive from the user. This makes prompt construction an important skill to develop. After you've mastered

the introduction, check out our article on more advanced prompt engineering techniques.

Getting access to Azure OpenAI Service

How do I get access to Azure OpenAI?

Access is currently limited as we navigate high demand, upcoming product improvements, and Microsoft's commitment to responsible AI. For now, we're working with customers with an existing partnership with Microsoft, lower risk use cases, and those committed to incorporating mitigations. In addition to applying for initial access, all solutions using Azure OpenAI are required to go through a use case review before they can be released for production use. Apply here for initial access or for a production review: Apply now

After I apply for access, how long will I have to wait to get approved?

We don't currently provide a timeline for access approval.

Learning more and where to ask questions

Where can I read about the latest updates to Azure OpenAI?

For monthly updates, see our what's new page.

Where can I get training to get started learning and build my skills around Azure OpenAI?

Check out our introduction to Azure OpenAI training course.

Where can I post questions and see answers to other common questions?

We recommend posting questions on Microsoft Q&A

Alternatively, you can post questions on Stack Overflow

Where do I go for Azure OpenAI customer support?

Azure OpenAI is part of Azure Cognitive Services. You can learn about all the support options for Azure Cognitive Services in the support and help options guide.

Models and fine-tuning

What models are available?

Consult the Azure OpenAI model availability guide.

Where can I find out what region a model is available in?

Consult the Azure OpenAI model availability guide for region availability.

What is the difference between a base model and a fine-tuned model?

A base model is a model that hasn't been customized or fine-tuned for a specific use case. Fine-tuned models are customized versions of base models where a model's weights are trained on a unique set of prompts. Fine-tuned models let you achieve better results on a wider number of tasks without needing to provide detailed examples for in-context learning as part of your completion prompt. To learn more, review our fine-tuning guide.

What is the maximum number of fine-tuned models I can create?

100

What are the SLAs for API responses in Azure OpenAI?

We don't have a defined API response time Service Level Agreement (SLA) at this time. The overall SLA for Azure OpenAI Service is the same as for other Azure Cognitive Services. For more information, see the Cognitive Services section of the Service Level Agreements (SLA) for Online Services page .

Why was my fine-tuned model deployment deleted?

If a customized (fine-tuned) model is deployed for more than fifteen (15) days during which no completions or chat completions calls are made to it, the deployment will automatically be deleted (and no further hosting charges will be incurred for that deployment). The underlying customized model will remain available and can be redeployed at any time. To learn more check out the how-to-article.

How do I deploy a model with the REST API?

There are currently two different REST APIs that allow model deployment. For the latest model deployment features such as the ability to specify a model version during deployment for models like text-embedding-ada-002 Version 2, use the Cognitive Services Create or Update REST API call.

Next steps

Azure OpenAI quotas and limits Azure OpenAI what's new Azure OpenAI quickstarts

Quickstart: Get started generating text using Azure OpenAI Service

Article • 05/24/2023

Use this article to get started making your first calls to Azure OpenAI.

Prerequisites

An Azure subscription - Create one for free .

Access granted to Azure OpenAI in the desired Azure subscription.

Currently, access to this service is granted only by application. You can apply for access to Azure OpenAI by completing the form at https://aka.ms/oai/access . Open an issue on this repo to contact us if you have an issue.

An Azure OpenAI resource with a model deployed. For more information about model deployment, see the resource deployment guide.

I ran into an issue with the prerequisites.

Go to the Azure OpenAI Studio

Navigate to Azure OpenAI Studio at https://oai.azure.com/ and sign-in with credentials that have access to your OpenAI resource. During or after the sign-in workflow, select the appropriate directory, Azure subscription, and Azure OpenAI resource.

From the Azure OpenAI Studio landing page navigate further to explore examples for prompt completion, manage your deployments and models, and find learning resources such as documentation and community forums.



Go to the Playground for experimentation and fine-tuning workflow.

Playground

Start exploring Azure OpenAI capabilities with a no-code approach through the GPT-3 Playground. It's simply a text box where you can submit a prompt to generate a completion. From this page, you can quickly iterate and experiment with the capabilities.



You can select a deployment and choose from a few pre-loaded examples to get started. If your resource doesn't have a deployment, select Create a deployment and follow the instructions provided by the wizard. For more information about model deployment, see the resource deployment guide.

You can experiment with the configuration settings such as temperature and pre-response text to improve the performance of your task. You can read more about each parameter in the REST API.

Selecting the Generate button will send the entered text to the completions API and stream the results back to the text box.

Select the Undo button to undo the prior generation call.

Select the Regenerate button to complete an undo and generation call together.

Azure OpenAI also performs content moderation on the prompt inputs and generated outputs. The prompts or responses may be filtered if harmful content is detected. For more information, see the content filter article.

In the GPT-3 playground you can also view Python and curl code samples pre-filled according to your selected settings. Just select View code next to the examples dropdown. You can write an application to complete the same task with the OpenAI Python SDK, curl, or other REST API client.

Try text summarization

To use the Azure OpenAI for text summarization in the GPT-3 Playground, follow these steps:

1. Sign in to Azure OpenAI Studio .

   
   

2. Select the subscription and OpenAI resource to work with.

   
   

3. Select GPT-3 Playground at the top of the landing page.

   
   

4. Select your deployment from the Deployments dropdown. If your resource doesn't have a deployment, select Create a deployment and then revisit this step.

5. Select Summarize Text from the Examples dropdown.

   

   
   

6. Select Generate . Azure OpenAI will attempt to capture the context of text and rephrase it succinctly. You should get a result that resembles the following text:

The accuracy of the response can vary per model. The Davinci based model in this example is well-suited to this type of summarization, whereas a Codex based model wouldn't perform as well at this particular task.

I ran into an issue with the playground.

Clean up resources

If you want to clean up and remove an OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Portal Azure CLI

Next steps

Learn more about how to generate the best completion in our How-to guide on completions.

For more examples check out the Azure OpenAI Samples GitHub repository .

Quickstart: Get started using ChatGPT and GPT-4 with Azure OpenAI Service

Article • 05/24/2023

Use this article to get started using Azure OpenAI.

Prerequisites

An Azure subscription - Create one for free .

Access granted to Azure OpenAI in the desired Azure subscription.

Currently, access to this service is granted only by application. You can apply for access to Azure OpenAI by completing the form at https://aka.ms/oai/access . Open an issue on this repo to contact us if you have an issue.

gpt-35-turbo

An Azure OpenAI Service resource with either the or the gpt-4 1

models deployed. For more information about model deployment, see the resource deployment guide.

1 GPT-4 models are currently only available by request. To access these models, existing Azure OpenAI customers can apply for access by filling out this form .

I ran into an issue with the prerequisites.

Go to Azure OpenAI Studio

Navigate to Azure OpenAI Studio at https://oai.azure.com/ and sign-in with credentials that have access to your OpenAI resource. During or after the sign-in workflow, select the appropriate directory, Azure subscription, and Azure OpenAI resource.

From the Azure OpenAI Studio landing page, select Chat playground.



Playground

Start exploring OpenAI capabilities with a no-code approach through the Azure OpenAI Studio Chat playground. From this page, you can quickly iterate and experiment with the capabilities.



Assistant setup

You can use the Assistant setup dropdown to select a few pre-loaded System message

examples to get started.

System messages give the model instructions about how it should behave and any context it should reference when generating a response. You can describe the assistant's personality, tell it what it should and shouldn't answer, and tell it how to format responses.

Add few-shot examples allows you to provide conversational examples that are used by the model for in-context learning.

At any time while using the Chat playground you can select View code to see Python, curl, and json code samples pre-populated based on your current chat session and settings selections. You can then take this code and write an application to complete the same task you're currently performing with the playground.

Chat session

Selecting the Send button sends the entered text to the completions API and the results are returned back to the text box.

Select the Clear chat button to delete the current conversation history.

Settings

Show panels

By default there are three panels: assistant setup, chat session, and settings. Show

panels allows you to add, remove, and rearrange the panels. If you ever close a panel and need to get it back, use Show panels to restore the lost panel.

Start a chat session

1. From the assistant setup drop-down, select Xbox customer support agent

   
   

2. You'll be prompted asking if you want to update the system message, select

   Continue.

   
   

3. In the chat session pane, enter the following question: "I'm interested in buying a new Xbox", and select Send.

4. You'll receive a response similar to:

   
   

   

   
   

5. Enter a follow-up question like: "which models support 4K?"

   

   
   

6. Now that you have a basic conversation select View code from under Assistant setup and you'll have a replay of the code behind the entire conversation so far:

I ran into an issue with the playground.

Understanding the prompt structure

If you examine the sample from View code you'll notice some unique tokens that weren't part of a typical GPT completion call. ChatGPT was trained to use special tokens to delineate different parts of the prompt. Content is provided to the model in between

<|im_start|>

<|im_end|>

and tokens. The prompt begins with a system message that

can be used to prime the model by including context or instructions for the model. After that, the prompt contains a series of messages between the user and the assistant.

The assistant's response to the prompt will then be returned below the

<|im_start|>assistant

<|im_end|>

token and will end with denoting that the assistant

has finished its response. You can also use the Show raw syntax toggle button to display these tokens within the chat session panel.

The ChatGPT how-to guide provides an in-depth introduction into the new prompt

gpt-35-turbo

structure and how to use the model effectively.

Clean up resources

Once you're done testing out the Chat playground, if you want to clean up and remove an OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Portal Azure CLI

Next steps

gpt-35-turbo

Learn more about how to work with ChatGPT and the new with the ChatGPT how-to guide.

model

For more examples check out the Azure OpenAI Samples GitHub repository

Content filtering

Article • 04/14/2023

Azure OpenAI Service includes a content management system that works alongside core models to filter content. This system works by running both the input prompt and generated content through an ensemble of classification models aimed at detecting misuse. If the system identifies harmful content, you'll receive either an error on the API call if the prompt was deemed inappropriate or the finish_reason on the response will

content_filter

be to signify that some of the generation was filtered. You can generate

content with the completions API using many different configurations that will alter the filtering behavior you should expect. The following section aims to enumerate all of these scenarios for you to appropriately design your solution.

To ensure you have properly mitigated risks in your application, you should evaluate all potential harms carefully, follow guidance in the Transparency Note and add scenario-specific mitigation as needed.

Scenario details

When building your application, you'll want to account for scenarios where the content returned by the Completions API is filtered and content may not be complete. How you act on this information will be application specific. The behavior can be summarized in the following key points:

Prompts that are deemed inappropriate will return an HTTP 400 error

Non-streaming completions calls won't return any content when the content is

finish_reason

filtered. The value will be set to content_filter . In rare cases with

finish_reason

long responses, a partial result can be returned. In these cases, the will be updated.

For streaming completions calls, segments will be returned back to the user as

they're completed. The service will continue streaming until either reaching a stop token, length or harmful content is detected.

Scenario: You send a non-streaming completions call asking for multiple generations with no inappropriate content

The table below outlines the various ways content filtering can appear:

Example request payload:

Example response JSON:

Scenario: Your API call asks for multiple responses (N>1) and at least 1 of the responses is filtered

Example request payload:

Example response JSON:

Scenario: An inappropriate input prompt is sent to the completions API (either for streaming or non-streaming)

Example request payload:

JSON

Example response JSON:

Scenario: You make a streaming completions call with all generated content passing the content filters

Example request payload:

Example response JSON:

Scenario: You make a streaming completions call asking for multiple generated responses and at least one response is filtered

Example request payload:

Example response JSON:

Scenario: Content filtering system doesn't run on the generation

Example request payload:

Example response JSON:

Best practices

As part of your application design you'll need to think carefully on how to maximize the benefits of your applications while minimizing the harms. Consider the following best practices:

How you want to handle scenarios where your users send in-appropriate or miss-use your application. Check the finish_reason to see if the generation is filtered.

error

If it's critical that the content filters run on your generations, check that there's no object in the content_filter_result .

user

user

To help with monitoring for possible misuse, applications serving multiple end-

users should pass the

parameter with each API call. The

should be a

unique identifier for the end-user. Don't send any actual user identifiable information as the value.

Next steps

Learn more about the underlying models that power Azure OpenAI.

Understanding embeddings in Azure OpenAI Service

Article • 05/10/2023

An embedding is a special format of data representation that can be easily utilized by machine learning models and algorithms. The embedding is an information dense representation of the semantic meaning of a piece of text. Each embedding is a vector of floating-point numbers, such that the distance between two embeddings in the vector space is correlated with semantic similarity between two inputs in the original format. For example, if two texts are similar, then their vector representations should also be similar.

Embedding models

Different Azure OpenAI embedding models are specifically created to be good at a particular task. Similarity embeddings are good at capturing semantic similarity between two or more pieces of text. Text search embeddings help measure whether long documents are relevant to a short query. Code search embeddings are useful for embedding code snippets and embedding natural language search queries.

Embeddings make it easier to do machine learning on large inputs representing words by capturing the semantic similarities in a vector space. Therefore, we can use embeddings to determine if two text chunks are semantically related or similar, and provide a score to assess similarity.

Cosine similarity

Azure OpenAI embeddings rely on cosine similarity to compute similarity between documents and a query.

From a mathematic perspective, cosine similarity measures the cosine of the angle between two vectors projected in a multi-dimensional space. This is beneficial because if two documents are far apart by Euclidean distance because of size, they could still have a smaller angle between them and therefore higher cosine similarity. For more information about cosine similarity equations, see this article on Wikipedia .

An alternative method of identifying similar documents is to count the number of common words between documents. Unfortunately, this approach doesn't scale since an expansion in document size is likely to lead to a greater number of common words

detected even among completely disparate topics. For this reason, cosine similarity can offer a more effective alternative.

Next steps

Learn more about using Azure OpenAI and embeddings to perform document search with our embeddings tutorial.

Introduction to red teaming large language models (LLMs)

Article • 05/19/2023

The term red teaming has historically described systematic adversarial attacks for testing security vulnerabilities. With the rise of LLMs, the term has extended beyond traditional cybersecurity and evolved in common usage to describe many kinds of probing, testing, and attacking of AI systems. With LLMs, both benign and adversarial usage can produce potentially harmful outputs, which can take many forms, including harmful content such as hate speech, incitement or glorification of violence, or sexual content.

Red teaming is an essential practice in the responsible development of systems and features using LLMs. While not a replacement for systematic measurement and mitigation work, red teamers help to uncover and identify harms and, in turn, enable measurement strategies to validate the effectiveness of mitigations.

Microsoft has conducted red teaming exercises and implemented safety systems (including content filters and other mitigation strategies) for its Azure OpenAI Service models (see this Responsible AI Overview). However, the context of your LLM application will be unique and you also should conduct red teaming to:

Test the LLM base model and determine whether there are gaps in the existing safety systems, given the context of your application system.

Identify and mitigate shortcomings in the existing default filters or mitigation strategies.

Provide feedback on failures so we can make improvements.

Here is how you can get started in your process of red teaming LLMs. Advance planning is critical to a productive red teaming exercise.

Getting started

Managing your red team

Assemble a diverse group of red teamers.

LLM red teamers should be a mix of people with diverse social and professional backgrounds, demographic groups, and interdisciplinary expertise that fits the deployment context of your AI system. For example, if you’re designing a chatbot to help health care providers, medical experts can help identify risks in that domain.

Recruit red teamers with both benign and adversarial mindsets.

Having red teamers with an adversarial mindset and security-testing experience is essential for understanding security risks, but red teamers who are ordinary users of your application system and haven’t been involved in its development can bring valuable perspectives on harms that regular users might encounter.

Remember that handling potentially harmful content can be mentally taxing.

You will need to take care of your red teamers, not only by limiting the amount of time they spend on an assignment, but also by letting them know they can opt out at any time. Also, avoid burnout by switching red teamers’ assignments to different focus areas.

Planning your red teaming

Where to test

Because a system is developed using a LLM base model, you may need to test at several different layers:

The LLM base model with its safety system in place to identify any gaps that may need to be addressed in the context of your application system. (Testing is usually through an API endpoint.)

Your application system. (Testing is usually through a UI.)

Both the LLM base model and your application system before and after mitigations are in place.

How to test

Consider conducting iterative red teaming in at least two phases:

1. Open-ended red teaming, where red teamers are encouraged to discover a variety of harms. This can help you develop a taxonomy of harms to guide further testing. Note that developing a taxonomy of undesired LLM outputs for your application system is crucial to being able to measure the success of specific mitigation efforts.

2. Guided red teaming, where red teamers are assigned to focus on specific harms

listed in the taxonomy while staying alert for any new harms that may emerge. Red teamers can also be instructed to focus testing on specific features of a system for surfacing potential harms.

Be sure to:

Provide your red teamers with clear instructions for what harms or system features they will be testing.

Give your red teamers a place for recording their findings. For example, this could be a simple spreadsheet specifying the types of data that red teamers should provide, including basics such as:

The type of harm that was surfaced.

The input prompt that triggered the output. An excerpt from the problematic output.

Comments about why the red teamer considered the output problematic.

Maximize the effort of responsible AI red teamers who have expertise for testing specific types of harms or undesired outputs. For example, have security subject matter experts focus on jailbreaks, metaprompt extraction, and content related to aiding cyberattacks.

Reporting red teaming findings

You will want to summarize and report red teaming top findings at regular intervals to key stakeholders, including teams involved in the measurement and mitigation of LLM failures so that the findings can inform critical decision making and prioritizations.

Next steps

Learn about other mitigation strategies like prompt engineering

Introduction to prompt engineering

Article • 03/21/2023

GPT-3, GPT-3.5, and GPT-4 models from OpenAI are prompt-based. With prompt-based models, the user interacts with the model by entering a text prompt, to which the model responds with a text completion. This completion is the model’s continuation of the input text.

While these models are extremely powerful, their behavior is also very sensitive to the prompt. This makes prompt construction an important skill to develop.

Prompt construction can be difficult. In practice, the prompt acts to configure the model weights to complete the desired task, but it's more of an art than a science, often requiring experience and intuition to craft a successful prompt. The goal of this article is to help get you started with this learning process. It attempts to capture general concepts and patterns that apply to all GPT models. However it's important to understand that each model behaves differently, so the learnings may not apply equally to all models.

Basics

This section covers the basic concepts and elements of GPT prompts.

Text prompts are how users interact with GPT models. As with all generative language models, GPT models attempt to produce the next series of words that are most likely to follow from the previous text. It's as if we're saying What is the first thing that comes to your mind when I say <prompt> ?”. The examples below demonstrate this behavior. Given the first words of famous content, the model is able to accurately continue the text.

Prompt Completion

Four score

and seven years ago our

"Call me Ishmael."

fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal. […]

"Some years ago—never mind how long precisely—having little o no money in my purse, and nothing particular to interest me on shore, I thought I would sail about a little and see the watery part of the world." […]

As you develop more complex prompts, it's helpful to keep this fundamental behavior in mind. Regardless of the prompt that is provided, the model is simply responding with what it determines is most likely (given its training data and training targets). If you ask

a question in your prompt, the model isn't following a separate “Q&A” code path, but rather it appears to answer the question because an answer is the most likely sort of response for the given question as input.

Prompt components

When using the Completion API while there's no differentiation between different parts of the prompt, it can still be useful for learning and discussion to identify underlying prompt components. With the Chat Completion API there are distinct sections of the prompt that are sent to the API in the form of an array of dictionaries with associated roles: system, user, and assistant. This guidance will focus more generally on how to think about prompt construction rather than providing prescriptive guidance that is specific to one API over another.

It's also important to understand that while there could be other valid ways to dissect prompts, the goal of this breakdown is to provide a relatively simple way to think about prompt construction. With the Completion API, all the components are optional, but at least one must be present and most prompts include more than one component. There can be some grey area between components as well. The order presented below roughly corresponds to how commonly each component is used, from most to least.

Instructions

Instructions are likely the most commonly used prompt component. Instructions are simply that: instructions to the model on what to do. While simple in concept, they can be complex in practice. The following table illustrates this by showing both simple and more complex instructions, in two different forms.

InstructionPrompt Completion Type

Simple

Welcome to our weekly newsletter. This week we

Write an introduction forhave a special focus on productivity. We'll be

instructionsa weekly newsletter.

sharing tips and tricks on how to be productive in all areas of your life. We hope you enjoy and please share with your friends!

InstructionPrompt Completion Type

Write an introduction for Welcome to Contoso's weekly newsletter! We had a

Complex

a weekly newsletter to my company named Contoso. It should mention the great all

great all hands meeting last week and I want to thank the team for all their hard work over the tough past few months. We're looking forward to a

bright future and I'm confident that with our team's

instructionshands meeting we had

last week and thank the team for all their hard work over the tough past few months.

Write an introduction for

a weekly newsletter, including the following:

dedication and commitment, we will achieve great things. Thank you for being a part of our team and I look forward to working with you all in the coming weeks and months. Sincerely, [Your name].

Hello Contoso team! I wanted to take a moment to

- My company is named thank you for all your hard work over the past few

Complex

contoso

- An exuberant greeting

months. It's been a tough couple of months, but you've all pulled together and done an amazing

Instructions- Thanking the team for

their hard work over a couple tough months

job. I'm really proud of you all. Looking ahead to the next quarter, I'm feeling really positive. I think we're going to achieve some great things. Thanks

• Positive outlook for the again, The Senior Leadership Team. coming quarter

• Signed by the SLT

Primary Content

Primary content refers to some sort of text that is being processed or transformed by the model. Primary content is typically used in conjunction with instructions. A simple example would be language translation. In the example below, the English text would be considered the primary content while “Translate to French:” is an instruction.

Prompt Completion

Pouvez-vous s'il vous plaît me dire comment

aller au musée?

Can you please tell me how to get to the museum?

Translate to French:

Primary content can also be much longer. In the following example the primary content is the introduction section from the Wikipedia entry on time complexity , which is almost 300 words long. This content has been abbreviated in our table for ease of display.

Prompt Completion

In computer science, the time complexity is the

computational complexity that describes the amount of computer time it takes to run

Time complexity is a measure of how long an algorithm takes to run. It is typically expressed as a function of the size of the input. The most common measure is worst-

an algorithm. Time complexitycase time complexity, which is the maximum amount of is commonly estimated by [...] time required for inputs of a given size.

Summarize the previous text in a few sentences:

In computer science, the time The time complexity of an algorithm is the amount of

complexity is the computational complexity that describes the amount of computer time it takes to run

time it takes to run an algorithm. Time complexity is commonly estimated by counting the number of elementary operations performed by the algorithm,

supposing that each elementary operation takes a fixed

an algorithm. Time complexityamount of time to perform. The time complexity is

is commonly estimated by [...] generally expressed as a function of the size of the input. TL;DR (Too long didn't read)

GPT models can also handle primary content that is structured. In the example below, a TSV about beers in Belgium (abbreviated for display in this article) is passed to the model as part of the prompt. It is capable of properly interpreting the content and answering questions about the data.

Prompt Completion

Belgian Beer Brands

Beer name Beer style color ABV Brewery

"Chimay Gold: ""Doree""" trappist pale ale gold

4.80% Chimay Brewery (official trappist)

"Chimay Blue: ""Grande Reserve""" trappist dark ale

dark 9.00%

There are 10 beers in the table that are less than 6% ABV.

[…]

Referencing the table above, how many beers are less than 6% ABV?

Examples

Successful prompts often rely on the practice of “one-shot” or “few-shot” learning. This refers to the inclusion of one or more examples of the desired behavior of the model, typically by including input and output pairs. This is not learning in the sense that the model is permanently changed, but rather that the examples better condition the model

to respond as desired for only the current inference. The use of prompts with no examples is sometimes referred to as “zero-shot” learning. Please note that with the Chat Completion API few-shot learning examples are typically added to the messages array in the form of example user/assistant interactions after the initial system message.

LearningPrompt Completion Type

Zero-shot

Headline: Coach confident injury won't derail Warriors

Topic:

Headline: Twins' Correa to use opt-out, test free agency

Topic: Baseball Headline: Qatar World Cup to have zones for sobering up

The coach is confident that the injury won't derail the Warriors' season. The team is still focused on their goals and that they will continue to work hard to achieve them.

Few-shotTopic: Soccer

Headline: Yates: Fantasy football intel for Week 6 Topic: Football Headline: Coach confident injury won't derail Warriors

Topic:

Basketball

The example above illustrates the utility of few-shot learning. Without the examples, the model seems to be guessing at the desired behavior, while the examples cleanly show the model how to operate. This also demonstrates the power of the model, it can infer the category of label that is wanted, even without a ‘basketball’ label in the examples.

Cue

Cues act as the “jumpstart” for the output of the model, helping to direct the model to the desired output. It is often a prefix that the model can build onto. In the example

below, we show the cue directing the story to include key elements. Cues are often used along with instructions, but not always. In the examples below, the model is being used to summarize a long announcement email. A cue is used in one case to help focus the output of the model and to suggest a specific output format (bullet points) in the other.

In the context of the Chat Completion API instructions take the form of the system message, and few-shot examples are indicated by providing a series of example user/assistant interactions to help prime the model.

NumberPrompt Completion of Cues

Performance reviews are a core component of our

1. Performance & Development

   approach at Contoso. […] Summarize the above email message: Performance reviews are a core component of our Performance & Development

2. approach at Contoso. […] Summarize the above email message:

   The key takeaway of the message is

   
   

   The purpose of this email is to inform employees of recent changes to the performance review experience, which is part of Contoso's Performance & Development approach. The changes include an improved structure, upgraded in-tool user experience, and focused prompts. Additionally, there is now an optional Conversation Starters section for employees to flag topics they'd like to discuss with their managers.

   
   

   that Contoso is improving the performance review experience by making it more focused on core priorities, expectations and impact. Additionally, they are upgrading the in-tool user experience with tips, resources and guidance. Finally, they are adding guidance on how to activate company-wide core priorities.

   NumberPrompt Completion of Cues

   Performance reviews are a core component of our Performance &

3. Development approach at

Contoso. […] Summarize the above email message:

Key Points:

•

Contoso is introducing changes to the performance review experience, based on feedback from employees.

• The changes include a new focus on core priorities, expectations, and impact; an upgraded user experience; and new guidance on company-wide core priorities.

• Performance reviews are a time to reflect on past performance and learning as well as to plan for future priorities and growth opportunities.

• Employees and managers should write comments in advance of the performance review meeting.

• There should be at least two performance reviews per year.

• The time investment for each review should remain about the same.

Supporting content

Supporting content is information that the model can utilize to influence the output in some way. It differs from primary content in that it is not the main target of task, but it is typically used along with primary content. Common examples include contextual information such as the current date, the name of the user, user preferences, and so on. The example below uses supporting content to help organize a set of planned workshops for the user. Without the supporting (important topics) the model simply lists off the workshops (truncated for display), when informed of my important topics, the model is able to accurately group the meetings.

SupportingPrompt Completion

Content

SupportingPrompt Completion

Content

Without

Workshop list:

Title Description

Planning Workshop Kickoff Please join us to kick off our workshop! Hear from our leaders about cross group priorities and learnings for the year ahead. The invitation to this event has already been sent to everyone in […]

Summarize the workshops above, grouped by topic:

Workshop list: Planning Workshop Kickoff

-Planning Workshop Wrap-up

• Core MeetUp

• Customer Driven Perspectives

  -Impact and Key Focus Areas

  -Best practices

• 
  

Accelerating app adoption […]

SupportingPrompt Completion

Content

With

Workshop list:

Title Description

Planning Workshop Kickoff Please join us to kick off our workshop! Hear from our leaders about cross group priorities and learnings for the year ahead. The invitation to this event has already been sent to everyone in […]

My Important Topics: prompt engineering, search, GPT models

Summarize the workshops above, grouped by my important topics:

1. Prompt Engineering:

   • New prompt engineering tips

   • The art of

     prompt engineering

2. Search:

   -Intro to vector search with embedding

3. GPT

   Models:

   • Intro to GPT-4

   • ChatGPT in-depth.

Best practices

Be Specific. Leave as little to interpretation as possible. Restrict the operational space.

Be Descriptive. Use analogies.

Double Down. Sometimes you may need to repeat yourself to the model. Give instructions before and after your primary content, use an instruction and a cue, etc.

Order Matters. The order in which you present information to the model may impact the output. Whether you put instructions before your content (“summarize the following…”) or after (“summarize the above…”) can make a difference in output. Even the order of few-shot examples can matter. This is referred to as recency bias.

Give the model an “out”. It can sometimes be helpful to give the model an alternative path if it is unable to complete the assigned task. For example, when asking a question over a piece of text you might include something like "respond

with ‘not found’ if the answer is not present". This can help the model avoid generating false responses.

Space efficiency

While the input size increases with each new generation of GPT models, there will continue to be scenarios that provide more data than the model can handle. GPT models break words into “tokens”. While common multi-syllable words are often a single token, less common words are broken in syllables. Tokens can sometimes be counter-intuitive, as shown by the example below which demonstrates token boundaries for different date formats. In this case, spelling out the entire month is more space efficient than a fully numeric date. The current range of token support goes from 2000 tokens with earlier GPT-3 models to up to 32,768 tokens with the 32k version of the latest GPT-4 model.

Given this limited space, it is important to use it as efficiently as possible.

Tables – As shown in the examples in the previous section, GPT models can understand tabular formatted data quite easily. This can be a space efficient way to include data, rather than preceding every field with name (such as with JSON).

White Space – Consecutive whitespaces are treated as separate tokens which can be an easy way to waste space. Spaces preceding a word, on the other hand, are typically treated as part of the same token as the word. Carefully watch your usage of whitespace and don’t use punctuation when a space alone will do.

Next steps

Learn more about Azure OpenAI

Prompt engineering techniques

Article • 04/23/2023

This guide will walk you through some advanced techniques in prompt design and prompt engineering. If you're new to prompt engineering, we recommend starting with our introduction to prompt engineering guide.

While the principles of prompt engineering can be generalized across many different model types, certain models expect a specialized prompt structure. For Azure OpenAI GPT models, there are currently two distinct APIs where prompt engineering comes into play:

Chat Completion API. Completion API.

Each API requires input data to be formatted differently, which in turn impacts overall prompt design. The Chat Completion API supports the ChatGPT and GPT-4 models. These models are designed to take input formatted in a specific chat-like transcript stored inside an array of dictionaries.

The Completion API supports the older GPT-3 models and has much more flexible input requirements in that it takes a string of text with no specific format rules. Technically the ChatGPT models can be used with either APIs, but we strongly recommend using the Chat Completion API for these models. To learn more, please consult our in-depth guide on using these APIs.

The techniques in this guide will teach you strategies for increasing the accuracy and grounding of responses you generate with a Large Language Model (LLM). It is, however, important to remember that even when using prompt engineering effectively you still need to validate the responses the models generate. Just because a carefully crafted prompt worked well for a particular scenario doesn't necessarily mean it will generalize more broadly to certain use cases. Understanding the limitations of LLMs, is just as important as understanding how to leverage their strengths.

This guide does not go in-depth into the mechanics behind the message structure for Chat Completions. If you aren't familiar with interacting with the ChatGPT and GPT-4 models programmatically we recommend reading our how-to guide on the Chat Completion API first.

All of the examples in this section of the guide were tested against a base GPT-4 model in English. If you are reading a localized version of this article in another language, these responses represent a localized translation of the English results. To learn more about potential limitations depending on what language you are using to prompt a model, please consult our Responsible AI transparency note.

System message

The system message is included at the beginning of the prompt and is used to prime the model with context, instructions, or other information relevant to your use case. You can use the system message to describe the assistant’s personality, define what the model should and shouldn’t answer, and define the format of model responses.

The example below, shows a sample system message and the resulting model response:

Some other examples of system messages are:

“Assistant is a large language model trained by OpenAI.”

“Assistant is an intelligent chatbot designed to help users answer technical questions about Azure OpenAI Service. Only answer questions using the context below and if you're not sure of an answer, you can say "I don't know".

“Assistant is an intelligent chatbot designed to help users answer their tax related questions.”

“You are an assistant designed to extract entities from text. Users will paste in a string of text and you will respond with entities you've extracted from the text as a

JSON object. Here's an example of your output format:

An important detail to understand is that even if you instruct a model in the system message to answer I don't know when unsure of an answer this does not guarantee that the request will be honored. A well designed system message can increase the likelihood of a certain outcome, but it is still possible that an incorrect response could be generated that contradicts the intent of the instruction in the system message.

Few-shot learning

A common way to adapt language models to new tasks is to use few-shot learning. In few-shot learning, a set of training examples is provided as part of the prompt to give additional context to the model.

When using the Chat Completions API, a series of messages between the User and Assistant (written in the new prompt format), can serve as examples for few-shot learning. These examples can be used to prime the model to respond in a certain way, emulate particular behaviors, and seed answers to common questions.

The table above describes the basic prompt structure, but for a detailed explanation of the exact prompt format you will need to refer to the Chat Completion guide.

Non chat scenarios

While the Chat Completion API is optimized to work with multi-turn conversations, it also can be used for non chat scenarios. For example, for a sentiment analysis scenario, you might use the following prompt:

Start with clear instructions

The sequence in which information appears in the prompt matters. This is because GPT style models are built in a certain way that defines how they process the input. Our research suggests that telling the model the task you want it to do at the beginning of the prompt, before sharing additional contextual information or examples, can help produce higher-quality outputs.

７ Note

Although following this technique is still generally recommended, in contrast to previous model versions (GPT-3 and prior), our testing showed that the model

response with ChatGPT and GPT-4 models was the same regardless of whether the technique is utilized. In the case of the below example, we see that adding the statement “Several sources … eruption” at the beginning or the end of the prompt did not cause any change in the final model response.

Repeat instructions at the end

Models can be susceptible to recency bias, which in this context means that information at the end of the prompt might have more significant influence over the output than information at the beginning of the prompt. Therefore, it is worth experimenting with repeating the instructions at the end of the prompt and evaluating the impact on the generated response.

Prime the output

“Here’s

This refers to including a few words or phrases at the end of the prompt to obtain a model response that follows the desired form. For example, using a cue such as

a bulleted list of key points:\n- ”

can help make sure the output is formatted as a

list of bullet points.

In the above prompt, the text One possible search query is: primes the model to produce a single output. Without this cue the model produces several search queries as output.

Add clear syntax

Using clear syntax for your prompt—including punctuation, headings, and section markers—helps communicate intent and often makes outputs easier to parse.

In the example below, separators ( --- in this case) have been added between different

---

sources of information or steps. This allows the use of as a stopping condition for

generation. In addition, section headings or special variables are presented in uppercase to differentiate them.

If you're not sure what syntax to use, consider using Markdown or XML. The models have been trained on a lot of web content in XML and Markdown which may provide better results.

Break the task down

Large language models (LLMs) often perform better if the task is broken down into smaller steps. For example, in the search query prompt referenced earlier, the prompt can be restructured so that the model is first instructed to extract relevant facts, and then instructed to generate search queries that can be used to verify those facts.

Notice the use of clear syntax to differentiate the sections and prime the output. In this simple example, breaking the task down from one to two steps is not very dramatic, but when trying to do this for a larger piece of text with many factual claims, breaking the task down can make a significant difference.

Use of affordances

Sometimes we can get the model to use an affordance instead of relying on its own parameters for information and answers. Search, for example, can be an affordance to help mitigate against fabricated answers, and to get up-to-date information.

One simple way to use an affordance is to stop generation once the affordance calls are generated by the model, then paste the outcomes back into the prompt. Below is an example of what the follow-up call could be after doing the SEARCH calls above. Notice how we pasted the search results into the prompt and replaced the previous SEARCH calls.

Chain of thought prompting

This is a variation on the break the task down technique. Instead of splitting a task into smaller steps, in this approach the model response is instructed to proceed step-by-step and present all the steps involved. Doing so reduces the possibility of inaccuracy of outcomes and makes assessing the model response easier.

Specifying the output structure

Using your prompt to specify the structure of the output can have a significant impact on the nature and quality of the results. Sometimes system message inputs as “only write true facts” or “do not fabricate information” might not be adequate mitigations. Instead, asking for the model response to also include citations can help result in a reduction of the prevalence of incorrect responses.

If you instruct the model to cite the source material when it makes statements, those statements are much more likely to be grounded. Asking for citations makes it so that

the model must make two errors every time it generates a response: the first error is the fabricated response, and the second is the bad citation. Note that the closer the citation is to the text it supports, the shorter the distance ahead the model needs to anticipate the citation, which suggests that inline citations are better mitigations for false content generation than citations at the end of the content.

Similarly, if you ask the model to extract factual statements from a paragraph, it may extract compound statements such as ‘X is doing Y AND Z’ (which may be harder to verify). This can be avoided by specifying an output structure such as (entity1, relationship, entity2).

The examples below show the use of citations and directing the model response to fit a defined structure.

Temperature and Top_p parameters

Changing the temperature parameter changes the output of the model. The temperature parameter can be set between 0 and 2. A higher value, for example 0.7 will make the output more random and produce more divergent responses, while a lower value, like. 0.2, will make the output more focused and concrete. A fictional story could be generated using a higher temperature. Whereas to generate a legal document it is

recommended to use a much lower temperature. Top_probability is another parameter and is similar to Temperature in that it also controls the randomness of the model response, but it does so in a different way. The general recommendation is to alter one of these two parameters at a time, not both.

Provide grounding context

One of the most effective ways to provide reliable answers is to give the model data to draw its responses from (grounding data). If your use case relies on up-to-date, reliable information and is not purely a creative scenario, we strongly recommend providing grounding data. In general, the closer you can get your source material to the final form of the answer you want, the less work the model needs to do, which means there is less opportunity for error. Here is an example where the system is provided a recent blog describing the launch of GPT-4 in Azure OpenAI Service , and asked to name some of its early customers.

1 Text from Azure OpenAI GPT-4 launch blog .

Next steps

Learn more about Azure OpenAI.

Get started with the ChatGPT model with the ChatGPT quickstart.

For more examples, check out the Azure OpenAI Samples GitHub repository

System message framework and template recommendations for Large Language Models (LLMs)

Article • 05/23/2023

This article provides a recommended framework and example templates to help write an effective system message, sometimes referred to as a metaprompt or system prompt that can be used to guide an AI system’s behavior and improve system performance. If you're new to prompt engineering, we recommend starting with our introduction to prompt engineering and prompt engineering techniques guidance.

This guide provides system message recommendations and resources that, along with other prompt engineering techniques, can help increase the accuracy and grounding of responses you generate with a Large Language Model (LLM). However, it is important to remember that even when using these templates and guidance, you still need to validate the responses the models generate. Just because a carefully crafted system message worked well for a particular scenario doesn't necessarily mean it will work more broadly across other scenarios. Understanding the limitations of LLMs and the mechanisms for evaluating and mitigating those limitations is just as important as understanding how to leverage their strengths.

The LLM system message framework described here covers four concepts:

Define the model’s profile, capabilities, and limitations for your scenario Define the model’s output format

Provide example(s) to demonstrate the intended behavior of the model Provide additional behavioral guardrails

Define the model’s profile, capabilities, and limitations for your scenario

Define the specific task(s) you would like the model to complete. Describe who the users of the model will be, what inputs they will provide to the model, and what you expect the model to do with the inputs.

Define how the model should complete the tasks, including any additional tools (like APIs, code, plug-ins) the model can use. If it doesn’t use additional tools, it can rely on its own parametric knowledge.

Define the scope and limitations of the model’s performance. Provide clear instructions on how the model should respond when faced with any limitations. For example, define how the model should respond if prompted on subjects or for uses that are off topic or otherwise outside of what you want the system to do.

Define the posture and tone the model should exhibit in its responses.

Define the model's output format

When using the system message to define the model’s desired output format in your scenario, consider and include the following types of information:

Define the language and syntax of the output format. If you want the output to be machine parse-able, you may want the output to be in formats like JSON, XSON or XML.

Define any styling or formatting preferences for better user or machine readability. For example, you may want relevant parts of the response to be bolded or citations to be in a specific format.

Provide example(s) to demonstrate the intended behavior of the model

When using the system message to demonstrate the intended behavior of the model in your scenario, it is helpful to provide specific examples. When providing examples, consider the following:

Describe difficult use cases where the prompt is ambiguous or complicated, to give the model additional visibility into how to approach such cases.

Show the potential “inner monologue” and chain-of-thought reasoning to better inform the model on the steps it should take to achieve the desired outcomes.

Define additional behavioral guardrails

When defining additional safety and behavioral guardrails, it’s helpful to first identify and prioritize the harms you’d like to address. Depending on the application, the sensitivity and severity of certain harms could be more important than others. Below, we’ve outlined some system message templates that may help mitigate some of the common harms that have been seen with LLMs, such as fabrication of content (that is not grounded or relevant), jailbreaks, and manipulation.

Next steps

Learn more about Azure OpenAI

Learn more about deploying Azure OpenAI responsibly

For more examples, check out the Azure OpenAI Samples GitHub repository

Create a resource and deploy a model using Azure OpenAI

Article • 03/08/2023 • 5 minutes to read

Use this article to get started with Azure OpenAI with step-by-step instructions to create a resource and deploy a model. While the steps for resource creation and model deployment can be completed in a few minutes, the actual deployment process itself can take more than hour. You can create your resource, start your deployment, and then check back in on your deployment later rather than actively waiting for the deployment to complete.

Prerequisites

An Azure subscription - Create one for free

Access granted to Azure OpenAI in the desired Azure subscription

Currently, access to this service is granted only by application. You can apply for access to Azure OpenAI by completing the form at https://aka.ms/oai/access . Open an issue on this repo to contact us if you have an issue.

Create a resource

Resources in Azure can be created several different ways: Within the Azure portal

Using the REST APIs, Azure CLI, PowerShell or client libraries Via ARM templates

This guide walks you through the Azure portal creation experience.

1. Navigate to the create page: Azure OpenAI Service Create Page

   
   

2. On the Create page provide the following information:



Deploy a model

Before you can generate text or inference, you need to deploy a model. You can select from one of several available models in Azure OpenAI Studio.

Davinci is the most capable model family and can perform any task the other models can perform and often with less instruction. For applications requiring a lot of understanding of the content, like summarization for a specific audience and creative content generation, Davinci is going to produce the best results.

To deploy a model, follow these steps:

1. Sign in to Azure OpenAI Studio .

   
   

2. Select the subscription and OpenAI resource to work with.

   
   

3. Select Manage deployments in your resource > Go to Deployments under Manage your deployments and models. You might first need to scroll down on the landing page.

   
   

   

   
   

4. Select Create new deployment from the Management > Deployments page.

   
   

   text-davinci-003

5. Select a model from the drop-down. For getting started in the East US region, we

   text-davinci-002

   recommend the the

   model. In other regions you should start with model. Some models are not available in all regions. For a

   list of available models per region, see Model Summary table and region availability.

6. Enter a model name to help you identify the model. Choose a name carefully. The model name will be used as the deployment name via OpenAI client libraries and API.

7. Select Create to deploy the model.

The deployments table displays a new entry that corresponds to this newly created model. Your deployment status will move to succeeded when the deployment is complete and ready for use.

Next steps

Now that you have a resource and your first model deployed get started making API calls and generating text with our quickstarts.

Learn more about the underlying models that power Azure OpenAI.

Learn how to work with the ChatGPT and GPT-4 models

Article • 05/15/2023

The ChatGPT and GPT-4 models are language models that are optimized for conversational interfaces. The models behave differently than the older GPT-3 models. Previous models were text-in and text-out, meaning they accepted a prompt string and returned a completion to append to the prompt. However, the ChatGPT and GPT-4 models are conversation-in and message-out. The models expect input formatted in a specific chat-like transcript format, and return a completion that represents a model-written message in the chat. While this format was designed specifically for multi-turn conversations, you'll find it can also work well for non-chat scenarios too.

In Azure OpenAI there are two different options for interacting with these type of models:

Chat Completion API.

Completion API with Chat Markup Language (ChatML).

The Chat Completion API is a new dedicated API for interacting with the ChatGPT and GPT-4 models. This API is the preferred method for accessing these models. It is also the only way to access the new GPT-4 models.

ChatML uses the same completion API that you use for other models like text-davinci-002, it requires a unique token based prompt format known as Chat Markup Language (ChatML). This provides lower level access than the dedicated Chat Completion API, but also requires additional input validation, only supports ChatGPT (gpt-35-turbo) models, and the underlying format is more likely to change over time.

This article walks you through getting started with the new ChatGPT and GPT-4 models. It's important to use the techniques described here to get the best results. If you try to interact with the models the same way you did with the older model series, the models will often be verbose and provide less useful responses.

Working with the ChatGPT and GPT-4 models

The following code snippet shows the most basic way to use the ChatGPT and GPT-4 models with the Chat Completion API. If this is your first time using these models programmatically, we recommend starting with our ChatGPT & GPT-4 Quickstart.

GPT-4 models are currently only available by request. Existing Azure OpenAI customers can apply for access by filling out this form .

Output

７ Note

The following parameters aren't available with the new ChatGPT and GPT-4 models:

logprobs , best_of , and echo . If you set any of these parameters, you'll get an error.

finish_reason

Every response includes a finish_reason . The possible values for are:

stop: API returned complete model output.

length: Incomplete model output due to max_tokens parameter or token limit.

content_filter: Omitted content due to a flag from our content filters.

null:API response still in progress or incomplete.

max_tokens

Consider setting to a slightly higher value than normal such as 300 or 500.

This ensures that the model doesn't stop generating text before it reaches the end of the message.

Model versioning

７ Note

gpt-35-turbo is equivalent to the gpt-3.5-turbo model from OpenAI.

gpt-35-turbo

gpt-

Unlike previous GPT-3 and GPT-3.5 models, the model as well as the

gpt-4-32k

4 and models will continue to be updated. When creating a deployment of

these models, you'll also need to specify a model version.

0301

Currently, only version

is available for ChatGPT and

for GPT-4 models. We'll

0314

continue to make updated versions available in the future. You can find model deprecation times on our models page.

Working with the Chat Completion API

OpenAI trained the ChatGPT and GPT-4 models to accept input formatted as a conversation. The messages parameter takes an array of dictionaries with a conversation organized by role.

The format of a basic Chat Completion is as follows:

A conversation with one example answer followed by a question would look like:

System role

The system role also known as the system message is included at the beginning of the array. This message provides the initial instructions to the model. You can provide various information in the system role including:

A brief description of the assistant Personality traits of the assistant

Instructions or rules you would like the assistant to follow

Data or information needed for the model, such as relevant questions from an FAQ

You can customize the system role for your use case or just include basic instructions. The system role/message is optional, but it's recommended to at least include a basic one to get the best results.

Messages

After the system role, you can include a series of messages between the user and the

assistant.

To trigger a response from the model, you should end with a user message indicating that it's the assistant's turn to respond. You can also include a series of example messages between the user and the assistant as a way to do few shot learning.

Message prompt examples

The following section shows examples of different styles of prompts that you could use with the ChatGPT and GPT-4 models. These examples are just a starting point, and you can experiment with different prompts to customize the behavior for your own use cases.

Basic example

If you want the ChatGPT model to behave similarly to chat.openai.com , you can use a basic system message like "Assistant is a large language model trained by OpenAI."

Example with instructions

For some scenarios, you may want to give additional instructions to the model to define guardrails for what the model is able to do.

Using data for grounding

You can also include relevant data or information in the system message to give the model extra context for the conversation. If you only need to include a small amount of information, you can hard code it in the system message. If you have a large amount of

data that the model should be aware of, you can use embeddings or a product like Azure Cognitive Search to retrieve the most relevant information at query time.

Few shot learning with Chat Completion

You can also give few shot examples to the model. The approach for few shot learning has changed slightly because of the new prompt format. You can now include a series of messages between the user and the assistant in the prompt as few shot examples. These examples can be used to seed answers to common questions to prime the model or teach particular behaviors to the model.

This is only one example of how you can use few shot learning with ChatGPT and GPT-4. You can experiment with different approaches to see what works best for your use case.

Using Chat Completion for non-chat scenarios

The Chat Completion API is designed to work with multi-turn conversations, but it also works well for non-chat scenarios.

For example, for an entity extraction scenario, you might use the following prompt:

Creating a basic conversation loop

The examples so far have shown you the basic mechanics of interacting with the Chat Completion API. This example shows you how to create a conversation loop that performs the following actions:

Continuously takes console input, and properly formats it as part of the messages array as user role content.

Outputs responses that are printed to the console and formatted and added to the messages array as assistant role content.

This means that every time a new question is asked, a running transcript of the conversation so far is sent along with the latest question. Since the model has no memory, you need to send an updated transcript with each new question or the model will lose context of the previous questions and answers.

When you run the code above you will get a blank console window. Enter your first question in the window and then hit enter. Once the response is returned, you can repeat the process and keep asking questions.

Managing conversations

The previous example will run until you hit the model's token limit. With each question

messages

gpt-

gpt-4

gpt-4-32k

asked, and answer received, the array grows in size. The token limit for

35-turbo

is 4096 tokens, whereas the token limits for

and

are 8192 and

max_tokens

32768 respectively. These limits include the token count from both the message array sent and the model response. The number of tokens in the messages array combined

with the value of the an error.

parameter must stay under these limits or you'll receive

It's your responsibility to ensure the prompt and completion falls within the token limit. This means that for longer conversations, you need to keep track of the token count and only send the model a prompt that falls within the limit.

The following code sample shows a simple chat loop example with a technique for handling a 4096 token count using OpenAI's tiktoken library.

pip install tiktoken

The code requires tiktoken 0.3.0 . If you have an older version run

--upgrade .

Python

del

pop()

In this example once the token count is reached the oldest messages in the conversation

transcript will be removed.

is used instead of

for efficiency, and we start at

index 1 so as to always preserve the system message and only remove user/assistant messages. Over time, this method of managing the conversation can cause the conversation quality to degrade as the model will gradually lose context of the earlier portions of the conversation.

An alternative approach is to limit the conversation duration to the max token length or a certain number of turns. Once the max token limit is reached and the model would lose context if you were to allow the conversation to continue, you can prompt the user that they need to begin a new conversation and clear the messages array to start a brand new conversation with the full token limit available.

The token counting portion of the code demonstrated previously, is a simplified version of one of OpenAI's cookbook examples .

Next steps

Learn more about Azure OpenAI.

Get started with the ChatGPT model with the ChatGPT quickstart.

For more examples, check out the Azure OpenAI Samples GitHub repository

Learn how to generate or manipulate text

Article • 02/17/2023 • 16 minutes to read

The completions endpoint can be used for a wide variety of tasks. It provides a simple but powerful text-in, text-out interface to any of our models. You input some text as a prompt, and the model will generate a text completion that attempts to match whatever context or pattern you gave it. For example, if you give the API the prompt, "As Descartes said, I think, therefore", it will return the completion " I am" with high probability.

The best way to start exploring completions is through our playground in Azure OpenAI Studio . It's a simple text box where you can submit a prompt to generate a completion. You can start with a simple example like the following:

write a tagline for an ice cream shop

once you submit, you'll see something like the following generated:

The actual completion results you see may differ because the API is stochastic by default. In other words, you might get a slightly different completion every time you call it, even if your prompt stays the same. You can control this behavior with the temperature setting.

This simple, "text in, text out" interface means you can "program" the model by providing instructions or just a few examples of what you'd like it to do. Its success generally depends on the complexity of the task and quality of your prompt. A general rule is to think about how you would write a word problem for a middle school student to solve. A well-written prompt provides enough information for the model to know what you want and how it should respond.

７ Note

Keep in mind that the models' training data cuts off in October 2019, so they may not have knowledge of current events. We plan to add more continuous training in the future.

Prompt design

Basics

OpenAI's models can do everything from generating original stories to performing complex text analysis. Because they can do so many things, you have to be explicit in showing what you want. Showing, not just telling, is often the secret to a good prompt.

The models try to predict what you want from the prompt. If you send the words "Give me a list of cat breeds," the model wouldn't automatically assume that you're asking for a list of cat breeds. You could as easily be asking the model to continue a conversation where the first words are "Give me a list of cat breeds" and the next ones are "and I'll tell you which ones I like." If the model only assumed that you wanted a list of cats, it wouldn't be as good at content creation, classification, or other tasks.

There are three basic guidelines to creating prompts:

Show and tell. Make it clear what you want either through instructions, examples, or a combination of the two. If you want the model to rank a list of items in alphabetical order or to classify a paragraph by sentiment, show it that's what you want.

Provide quality data. If you're trying to build a classifier or get the model to follow a pattern, make sure that there are enough examples. Be sure to proofread your examples

— the model is usually smart enough to see through basic spelling mistakes and give you a response, but it also might assume that the mistakes are intentional and it can affect the response.

Check your settings. The temperature and top_p settings control how deterministic the model is in generating a response. If you're asking it for a response where there's only one right answer, then you'd want to set these settings to lower values. If you're looking for a response that's not obvious, then you might want to set them to higher values. The number one mistake people use with these settings is assuming that they're "cleverness" or "creativity" controls.

Troubleshooting

If you're having trouble getting the API to perform as expected, follow this checklist:

1. Is it clear what the intended generation should be?

2. Are there enough examples?

3. Did you check your examples for mistakes? (The API won't tell you directly)

4. Are you using temp and top_p correctly?

Classification

To create a text classifier with the API we provide a description of the task and provide a few examples. In this demonstration we show the API how to classify the sentiment of Tweets.

It's worth paying attention to several features in this example:

1. Use plain language to describe your inputs and outputs We use plain language for the input "Tweet" and the expected output "Sentiment." For best practices, start with plain language descriptions. While you can often use shorthand or keys to indicate the input and output, when building your prompt it's best to start by being as descriptive as possible and then working backwards removing extra words as long as the performance to the prompt is consistent.

2. Show the API how to respond to any case In this example we provide multiple outcomes "Positive", "Negative" and "Neutral." A neutral outcome is important because there will be many cases where even a human would have a hard time determining if something is positive or negative and situations where it's neither.

3. You can use text and emoji The classifier is a mix of text and emoji 👍. The API reads emoji and can even convert expressions to and from them.

4. You need fewer examples for familiar tasks For this classifier we only provided a handful of examples. This is because the API already has an understanding of sentiment

   and the concept of a tweet. If you're building a classifier for something the API might not be familiar with, it might be necessary to provide more examples.

   
   

   Improving the classifier's efficiency

   Now that we have a grasp of how to build a classifier, let's take that example and make it even more efficient so that we can use it to get multiple results back from one API call.

   
   

   This is a tweet sentiment classifier Tweet: "I loved the new Batman movie!" Sentiment: Positive Tweet: "I hate it when my phone battery dies" Sentiment: Negative Tweet: "My day has been 👍" Sentiment: Positive Tweet: "This is the link to the article" Sentiment: Neutral Tweet text Tweet sentiment ratings: 1: Positive 2: Negative 3: Positive 4: Neutral 5: Positive Tweet text Tweet sentiment ratings: 1.

   1. "I loved the new Batman movie!"

   2. "I hate it when my phone battery dies"

   3. "My day has been 👍"

   4. "This is the link to the article"

   5. "This new music video blew my mind"

   1. "I can't stand homework"

   2. "This sucks. I'm bored 😠"

   3. "I can't wait for Halloween!!!"

   4. "My cat is adorable ❤❤"

   5. "I hate chocolate"

   After showing the API how tweets are classified by sentiment we then provide it a list of tweets and then a list of sentiment ratings with the same number index. The API is able

   to pick up from the first example how a tweet is supposed to be classified. In the second example it sees how to apply this to a list of tweets. This allows the API to rate five (and even more) tweets in just one API call.

   It's important to note that when you ask the API to create lists or evaluate text you need to pay extra attention to your probability settings (Top P or Temperature) to avoid drift.

   1. Make sure your probability setting is calibrated correctly by running multiple tests.

      
      

   2. Don't make your list too long or the API is likely to drift.

Generation

One of the most powerful yet simplest tasks you can accomplish with the API is generating new ideas or versions of input. You can give the API a list of a few story ideas and it will try to add to that list. We've seen it create business plans, character descriptions and marketing slogans just by providing it a handful of examples. In this demonstration we'll use the API to create more examples for how to use virtual reality in the classroom:

2.

All we had to do in this example is provide the API with just a description of what the list

is about and one example. We then prompted the API with the number that it's a continuation of the list.

indicating

Although this is a very simple prompt, there are several details worth noting:

1. We explained the intent of the list

   Just like with the classifier, we tell the API up front what the list is about. This helps it focus on completing the list and not trying to guess what the pattern is behind it.

2. Our example sets the pattern for the rest of the list

   Because we provided a one-sentence description, the API is going to try to follow that pattern for the rest of the items it adds to the list. If we want a more verbose response, we need to set that up from the start.

3. We prompt the API by adding an incomplete entry

2.

When the API sees and the prompt abruptly ends, the first thing it tries to do is

figure out what should come after it. Since we already had an example with number one and gave the list a title, the most obvious response is to continue adding items to the list.

Advanced generation techniques

You can improve the quality of the responses by making a longer more diverse list in your prompt. One way to do that is to start off with one example, let the API generate more and select the ones that you like best and add them to the list. A few more high-quality variations can dramatically improve the quality of the responses.

Conversation

The API is extremely adept at carrying on conversations with humans and even with itself. With just a few lines of instruction, we've seen the API perform as a customer service chatbot that intelligently answers questions without ever getting flustered or a wise-cracking conversation partner that makes jokes and puns. The key is to tell the API how it should behave and then provide a few examples.

Here's an example of the API playing the role of an AI answering questions:

This is all it takes to create a chatbot capable of carrying on a conversation. But underneath its simplicity there are several things going on that are worth paying attention to:

1. We tell the API the intent but we also tell it how to behave Just like the other prompts, we cue the API into what the example represents, but we also add another key detail: we give it explicit instructions on how to interact with the phrase "The assistant is helpful, creative, clever, and very friendly."

   Without that instruction the API might stray and mimic the human it's interacting with and become sarcastic or some other behavior we want to avoid.

2. We give the API an identity At the start we have the API respond as an AI that was created by OpenAI. While the API has no intrinsic identity, this helps it respond in a way that's as close to the truth as possible. You can use identity in other ways to create other kinds of chatbots. If you tell the API to respond as a woman who works as a research scientist in biology, you'll get intelligent and thoughtful comments from the API similar to what you'd expect from someone with that background.

In this example we create a chatbot that is a bit sarcastic and reluctantly answers questions:

To create an amusing and somewhat helpful chatbot we provide a few examples of questions and answers showing the API how to reply. All it takes is just a few sarcastic responses and the API is able to pick up the pattern and provide an endless number of snarky responses.

Transformation

The API is a language model that is familiar with a variety of ways that words and characters can be used to express information. This ranges from natural language text to code and languages other than English. The API is also able to understand content on a level that allows it to summarize, convert and express it in different ways.

Translation

In this example we show the API how to convert from English to French:

This example works because the API already has a grasp of French, so there's no need to try to teach it this language. Instead, we just need to provide enough examples that API understands that it's converting from one language to another.

If you want to translate from English to a language the API is unfamiliar with you'd need to provide it with more examples and a fine-tuned model to do it fluently.

Conversion

In this example we convert the name of a movie into emoji. This shows the adaptability of the API to picking up patterns and working with other characters.

Summarization

The API is able to grasp the context of text and rephrase it in different ways. In this example, the API takes a block of text and creates an explanation a child would understand. This illustrates that the API has a deep grasp of language.

In this example we place whatever we want summarized between the triple quotes. It's worth noting that we explain both before and after the text to be summarized what our intent is and who the target audience is for the summary. This is to keep the API from drifting after it processes a large block of text.

Completion

While all prompts result in completions, it can be helpful to think of text completion as its own task in instances where you want the API to pick up where you left off. For example, if given this prompt, the API will continue the train of thought about vertical farming. You can lower the temperature setting to keep the API more focused on the intent of the prompt or increase it to let it go off on a tangent.

This next prompt shows how you can use completion to help write React components. We send some code to the API, and it's able to continue the rest because it has an understanding of the React library. We recommend using models from our Codex series for tasks that involve understanding or generating code. Currently, we support two

code-davinci-002

Codex models: and code-cushman-001 . For more information about

Codex models, see the Codex models section in Models.

Factual responses

The API has a lot of knowledge that it's learned from the data it was trained on. It also has the ability to provide responses that sound very real but are in fact made up. There are two ways to limit the likelihood of the API making up an answer.

1. Provide a ground truth for the API If you provide the API with a body of text to answer questions about (like a Wikipedia entry) it will be less likely to confabulate a response.

2. Use a low probability and show the API how to say "I don't know" If the API understands that in cases where it's less certain about a response that saying "I don't know" or some variation is appropriate, it will be less inclined to make up answers.

In this example we give the API examples of questions and answers it knows and then examples of things it wouldn't know and provide question marks. We also set the probability to zero so the API is more likely to respond with a "?" if there's any doubt.

Working with code

The Codex model series is a descendant of OpenAI's base GPT-3 series that's been trained on both natural language and billions of lines of code. It's most capable in Python and proficient in over a dozen languages including C#, JavaScript, Go, Perl, PHP, Ruby, Swift, TypeScript, SQL, and even Shell.

Learn more about generating code completions, with the working with code guide

Next steps

Learn how to work with code (Codex). Learn more about the underlying models that power Azure OpenAI.

Additional resources

How to switch between OpenAI and Azure OpenAI endpoints with Python

Article • 05/26/2023

While OpenAI and Azure OpenAI Service rely on a common Python client library are small changes you need to make to your code in order to swap back and forth

, there

between endpoints. This article walks you through the common changes and differences you'll experience when working across OpenAI and Azure OpenAI.

７ Note

This library is maintained by OpenAI and is currently in preview. Refer to the release

history

library.

or the version.py commit history

to track the latest updates to the

Authentication

We recommend using environment variables. If you haven't done this before our Python quickstarts walk you through this configuration.

API key

Azure Active Directory authentication

Keyword argument for model

model

OpenAI uses the keyword argument to specify what model to use. Azure OpenAI

deployment_id

has the concept of deployments and uses the keyword argument to

engine

model

describe which model deployment to use. Azure OpenAI also supports the use of interchangeably with deployment_id .

engine

For OpenAI preferred.

still works in most instances, but it's deprecated and is

Azure OpenAI embeddings doesn't support multiple inputs

Many examples show passing multiple inputs into the embeddings API. For Azure OpenAI, currently we must pass a single text input per call.

Next steps

Learn more about how to work with ChatGPT and the GPT-4 models with our how-to guide.

For more examples, check out the Azure OpenAI Samples GitHub repository

Codex models and Azure OpenAI Service

Article • 02/17/2023 • 10 minutes to read

The Codex model series is a descendant of our GPT-3 series that's been trained on both natural language and billions of lines of code. It's most capable in Python and proficient in over a dozen languages including C#, JavaScript, Go, Perl, PHP, Ruby, Swift, TypeScript, SQL, and even Shell.

You can use Codex for a variety of tasks including:

Turn comments into code

Complete your next line or function in context

Bring knowledge to you, such as finding a useful library or API call for an application

Add comments

Rewrite code for efficiency

How to use the Codex models

Here are a few examples of using Codex that can be tested in Azure OpenAI Studio's playground with a deployment of a Codex series model, such as code-davinci-002 .

Saying "Hello" (Python)

Create random names (Python)

Create a MySQL query (Python)

Explaining code (JavaScript)

Best practices

Start with a comment, data or code

You can experiment using one of the Codex models in our playground (styling instructions as comments when needed.)

To get Codex to create a useful completion, it's helpful to think about what information a programmer would need to perform a task. This could simply be a clear comment or the data needed to write a useful function, like the names of variables or what class a function handles.

In this example we tell Codex what to call the function and what task it's going to perform.

This approach scales even to the point where you can provide Codex with a comment and an example of a database schema to get it to write useful query requests for various databases. Here's an example where we provide the columns and table names for the query.

When you show Codex the database schema, it's able to make an informed guess about how to format a query.

Specify the programming language

Codex understands dozens of different programming languages. Many share similar conventions for comments, functions and other programming syntax. By specifying the language and what version in a comment, Codex is better able to provide a completion for what you want. That said, Codex is fairly flexible with style and syntax. Here's an example for R and Python.

Prompt Codex with what you want it to do

If you want Codex to create a webpage, placing the first line of code in an HTML document ( <!DOCTYPE html> ) after your comment tells Codex what it should do next.

The same method works for creating a function from a comment (following the comment with a new line starting with func or def).

<!DOCTYPE html>

Placing it to do.

after our comment makes it very clear to Codex what we want

Or if we want to write a function we could start the prompt as follows and Codex will understand what it needs to do next.

Specifying libraries will help Codex understand what you want

Codex is aware of a large number of libraries, APIs and modules. By telling Codex which ones to use, either from a comment or importing them into your code, Codex will make suggestions based upon them instead of alternatives.

By specifying the version, you can make sure Codex uses the most current library.

７ Note

Codex can suggest helpful libraries and APIs, but always be sure to do your own research to make sure that they're safe for your application.

Comment style can affect code quality

With some languages, the style of comments can improve the quality of the output. For example, when working with Python, in some cases using doc strings (comments wrapped in triple quotes) can give higher quality results than using the pound ( # ) symbol.

Comments inside of functions can be helpful

Recommended coding standards usually suggest placing the description of a function inside the function. Using this format helps Codex more clearly understand what you want the function to do.

Provide examples for more precise results

If you have a particular style or format you need Codex to use, providing examples or demonstrating it in the first part of the request will help Codex more accurately match what you need.

Lower temperatures give more precise results

Setting the API temperature to 0, or close to zero (such as 0.1 or 0.2) tends to give better results in most cases. Unlike GPT-3 models, where a higher temperature can

provide useful creative and random results, higher temperatures with Codex models may give you really random or erratic responses.

In cases where you need Codex to provide different potential results, start at zero and then increment upwards by 0.1 until you find suitable variation.

Organize tasks into functions

We can get Codex to write functions by specifying what the function should do in as precise terms as possible in comment. For example, by writing the following comment, Codex creates a JavaScript timer function that's triggered when a user presses a button:

A simple JavaScript timer

Creating example data

Testing applications often requires using example data. Because Codex is a language model that understands how to comprehend and write natural language, you can ask Codex to create data like arrays of made up names, products and other variables. For example, here we ask Codex to create an array of weather temperatures.

Asking Codex to perform this task will produce a table like this:

Compound functions and small applications

We can provide Codex with a comment consisting of a complex request like creating a random name generator or performing tasks with user input and Codex can generate the rest provided there are enough tokens.

Limit completion size for more precise results or lower latency

Requesting longer completions in Codex can lead to imprecise answers and repetition. Limit the size of the query by reducing max_tokens and setting stop tokens. For

\n

instance, add as a stop sequence to limit completions to one line of code. Smaller

completions also incur less latency.

Use streaming to reduce latency

Large Codex queries can take tens of seconds to complete. To build applications that require lower latency, such as coding assistants that perform autocompletion, consider using streaming. Responses will be returned before the model finishes generating the entire completion. Applications that need only part of a completion can reduce latency by cutting off a completion either programmatically or by using creative values for stop .

Users can combine streaming with duplication to reduce latency by requesting more than one solution from the API, and using the first response returned. Do this by setting n > 1 . This approach consumes more token quota, so use carefully (for example, by

max_tokens

using reasonable settings for and stop ).

Use Codex to explain code

Codex's ability to create and understand code allows us to use it to perform tasks like explaining what the code in a file does. One way to accomplish this is by putting a comment after a function that starts with "This function" or "This application is." Codex will usually interpret this as the start of an explanation and complete the rest of the text.

Explaining an SQL query

In this example, we use Codex to explain in a human readable format what an SQL query is doing.

Writing unit tests

Creating a unit test can be accomplished in Python simply by adding the comment "Unit test" and starting a function.

Checking code for errors

By using examples, you can show Codex how to identify errors in code. In some cases no examples are required, however demonstrating the level and detail to provide a

description can help Codex understand what to look for and how to explain it. (A check by Codex for errors shouldn't replace careful review by the user. )

Using source data to write database functions

Just as a human programmer would benefit from understanding the database structure and the column names, Codex can use this data to help you write accurate query requests. In this example, we insert the schema for a database and tell Codex what to query the database for.

Converting between languages

You can get Codex to convert from one language to another by following a simple format where you list the language of the code you want to convert in a comment, followed by the code and then a comment with the language you want it translated into.

Rewriting code for a library or framework

If you want Codex to make a function more efficient, you can provide it with the code to rewrite followed by an instruction on what format to use.

Next steps

Learn more about the underlying models that power Azure OpenAI.

Additional resources

Documentation

How to customize a model with Azure OpenAI - Azure OpenAI

Learn how to create your own customized model with Azure OpenAI

How to generate text with Azure OpenAI - Azure OpenAI

Learn how to generate or manipulate text, including code with Azure OpenAI

How to prepare a dataset for custom model training - Azure OpenAI

Learn how to prepare your dataset for fine-tuning

Azure OpenAI models - Azure OpenAI

Learn about the different models that are available in Azure OpenAI.

Quickstart - Deploy a model and generate text using Azure OpenAI - Azure OpenAI

Walkthrough on how to get started with Azure OpenAI and make your first completions call.

Azure OpenAI embeddings tutorial - Azure OpenAI

Learn how to use the Azure OpenAI embeddings API for document search with the BillSum dataset

Azure OpenAI REST API reference - Azure OpenAI

Learn how to use the Azure OpenAI REST API. In this article, you'll learn about authorization options, how to structure a request and receive a response.

Azure OpenAI content filtering - Azure OpenAI

Learn about the content filtering capabilities of the OpenAI service in Azure Cognitive Services

Show 5 more

Learn how to generate embeddings with Azure OpenAI

Article • 04/20/2023

An embedding is a special format of data representation that can be easily utilized by machine learning models and algorithms. The embedding is an information dense representation of the semantic meaning of a piece of text. Each embedding is a vector of floating point numbers, such that the distance between two embeddings in the vector space is correlated with semantic similarity between two inputs in the original format. For example, if two texts are similar, then their vector representations should also be similar.

How to get embeddings

To obtain an embedding vector for a piece of text, we make a request to the embeddings endpoint as shown in the following code snippets:

console

curl

https://YOUR_RESOURCE_NAME.openai.azure.com/openai/deployments/YOUR_DEPL OYMENT_NAME/embeddings?api-version=2022-12-01\

-H 'Content-Type: application/json' \

-H 'api-key: YOUR_API_KEY' \

-d '{"input": "Sample Document goes here"}'

Console

Best Practices

Verify inputs don't exceed the maximum length

The maximum length of input text for our embedding models is 2048 tokens (equivalent to around 2-3 pages of text). You should verify that your inputs don't exceed this limit before making a request.

Choose the best model for your task

<search_model>-

doc

For the search models, you can obtain embeddings in two ways. The model is used for longer pieces of text (to be searched over) and the

<search_model>-query

model is used for shorter pieces of text, typically queries or class

labels in zero shot classification. You can read more about all of the Embeddings models in our Models guide.

Replace newlines with a single space

Unless you're embedding code, we suggest replacing newlines (\n) in your input with a single space, as we have observed inferior results when newlines are present.

Limitations & risks

Our embedding models may be unreliable or pose social risks in certain cases, and may cause harm in the absence of mitigations. Review our Responsible AI content for more information on how to approach their use responsibly.

Next steps

Learn more about using Azure OpenAI and embeddings to perform document search with our embeddings tutorial.

Learn more about the underlying models that power Azure OpenAI.

Learn how to prepare your dataset for fine-tuning

Article • 03/14/2023 • 11 minutes to read

The first step of customizing your model is to prepare a high quality dataset. To do this you'll need a set of training examples composed of single input prompts and the associated desired output ('completion'). This format is notably different than using models during inference in the following ways:

Only provide a single prompt vs a few examples.

You don't need to provide detailed instructions as part of the prompt.

Each prompt should end with a fixed separator to inform the model when the prompt ends and the completion begins. A simple separator, which generally works well is \n\n###\n\n . The separator shouldn't appear elsewhere in any prompt.

Each completion should start with a whitespace due to our tokenization, which tokenizes most words with a preceding whitespace.

Each completion should end with a fixed stop sequence to inform the model when the completion ends. A stop sequence could be \n , ### , or any other token that doesn't appear in any completion.

For inference, you should format your prompts in the same way as you did when creating the training dataset, including the same separator. Also specify the same stop sequence to properly truncate the completion.

The dataset cannot exceed 100 MB in total file size.

Best practices

Customization performs better with high-quality examples and the more you have, generally the better the model performs. We recommend that you provide at least a few hundred high-quality examples to achieve a model that performs better than using well-designed prompts with a base model. From there, performance tends to linearly increase with every doubling of the number of examples. Increasing the number of examples is usually the best and most reliable way of improving performance.

If you're fine-tuning on a pre-existing dataset rather than writing prompts from scratch, be sure to manually review your data for offensive or inaccurate content if possible, or review as many random samples of the dataset as possible if it's large.

Specific guidelines

Fine-tuning can solve various problems, and the optimal way to use it may depend on your specific use case. Below, we've listed the most common use cases for fine-tuning and corresponding guidelines.

Classification

Classifiers are the easiest models to get started with. For classification problems we suggest using ada, which generally tends to perform only very slightly worse than more capable models once fine-tuned, while being significantly faster. In classification problems, each prompt in the dataset should be classified into one of the predefined classes. For this type of problem, we recommend:

Use a separator at the end of the prompt, for example, \n\n###\n\n . Remember to also append this separator when you eventually make requests to your model.

Choose classes that map to a single token. At inference time, specify

max_tokens=1 since you only need the first token for classification.

Ensure that the prompt + completion doesn't exceed 2048 tokens, including the separator

Aim for at least 100 examples per class

To get class log probabilities, you can specify logprobs=5 (for five classes) when using your model

Ensure that the dataset used for fine-tuning is very similar in structure and type of task as what the model will be used for

Case study: Is the model making untrue statements?

Let's say you'd like to ensure that the text of the ads on your website mention the correct product and company. In other words, you want to ensure the model isn't making things up. You may want to fine-tune a classifier which filters out incorrect ads.

The dataset might look something like the following:

In the example above, we used a structured input containing the name of the company,

\nSupported:

the product, and the associated ad. As a separator we used which clearly

separated the prompt from the completion. With a sufficient number of examples, the separator you choose doesn't make much of a difference (usually less than 0.4%) as long as it doesn't appear within the prompt or the completion.

For this use case we fine-tuned an ada model since it is faster and cheaper, and the performance is comparable to larger models because it's a classification task.

Now we can query our model by making a Completion request.

yes

Which will return either or no .

Case study: Sentiment analysis

Let's say you'd like to get a degree to which a particular tweet is positive or negative. The dataset might look something like the following:

Once the model is fine-tuned, you can get back the log probabilities for the first

logprobs=2

completion token by setting on the completion request. The higher the

Console

probability for positive class, the higher the relative sentiment. Now we can query our model by making a Completion request.

Which will return: